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¢ Integrated modular avionics (IMA) principles are attractive 
for inclusion in spacecraft architectures. 
= Consolidates multiple functions to shared computing platforms. 
= Reduces spacecraft cost, weight, and design complexity. 
= Interchangeable components increases overall system maintainability — 
important for long duration missions! 
° The Avionics and Software (A&S) project 
= Funded by NASA’s Advanced Exploration Systems program. 
= Developing a flexible mission agnostic spacecraft architecture according 
to IMA principles. 
= NASA can minimize development time and cost by utilizing existing 
commercial technologies. 
= Matures promising technologies for use in flight projects. 5 
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° IMA Considerations in Networking 

= Requires network capable of accommodating traffic from multiple highly 
diverse systems (e.g. critical vs. non-critical) — potentially all from 
one shared computer platform. 

= Must prevent cascading faults b/w systems of differing criticalities 
connected to the same physical network. 

A\ Most avionic system failures result from ineffective fault containment and 
the resulting domino effect. 

= Some network technologies are better suited for certain tasks. 

= Applying the same technology everywhere traditionally results in undue 
expense and limited performance. 


Results in hybrid architectures with multiple technologies (e.g. NASA's 


LRO has MIL-STD-1553, SpaceWire, LVDS). 
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¢ Ethernet is promising 
= Inexpensive, widespread, and high speed = highly flexible. 
= Commonality promotes interchangeability between components. 
= Can augment with QoS enhancements for critical applications. 


= The A&S project considers Ethernet fundamental in the design of 
future manned spacecraft. 


° Integrated Power, Avionics, and Software Lees, 


= Flexible evaluation environment 
for hardware and software in 
simulated mission scenarios. 

= Realistic framework of vehicle 
subsystems connected via 
Ethernet backbone. 
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Ethernet in Space Programs 
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° Classical Ethernet characteristics 
= Event-driven communication — messages are only sent in response to 
environmental or internal events (asynchronous). 
= Best-effort paradigm — no guarantees regarding transmission time or 
successful message delivery. 


¢ Timing within an Ethernet network is not predictable. 


= Event-triggered = multiple frames will need 
to travel through the matrix simultaneously. 
— Usually supported by the switch fabric's parallel 
arrangement (space partitioning). 
= Collisions occur when frames are forwarded 
simultaneously to the same output port. 


= Arbitration is needed to regulate input to the 
switch fabric. 
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° What factors impact forwarding delay? 
= 1) Degree of contention, 2) arbitration method 
= Frequency/severity of conflicts is highly variable. 
° Contention limits throughput 
= Leads to buffer overflows and dropped frames. input queues 
= 58.6% with inout FIFOs under uniform traffic. 


= >80% with VOQs, crosspoint buffers, and better 


arbitration procedures (e.g. matrix, wavefront). 


¢ Modern advancements don’t address unpredictable timing. 
= E.g. VOQs eliminate head-of-line blocking, but still require arbitration. 


Requests 


Flight critical functions must operate in an entirely predictable manner and 


require a level of network determinism that classical Ethernet can’t provide. 
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Quality of Service (QoS): Methods for controlling bandwidth, latency, jitter, 
or data loss in mission-critical networks (@.g. prioritization, traffic shaping). 


° “Industrial Ethernet” (e.g. $100Mbit/s EtherNet/IP, PROFINET) 
= Replaces proprietary Fieldbus solutions on factory floor (e.g. machinery). 
= Modified w/ master/slave arch., I/O controllers, and bus or ring topology. 
= RT services through specialized HW and extra protocols around payload. 


° Rate-Constrained (e.g. ARINC 664P7-1, IEEE 802.1BA AVB) 


= Predetermined knowledge of traffic patterns (max size, | 
frequency) ensures upper bound on TX delays. ing 


= A priori agreement of network devices 


prevents buffer overflows in switch. 
= Latency 1-10ms, < 500us jitter, arbitration. -—@ <@ 
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° Time-Triggered Ethernet (SAE AS6802) 


= Uses specialized end systems and network switches (like AFDX). 


= Network planning tool allocates each device a finite transmission window. 
= Each slot is repeated sequentially to form a periodic comm. schedule. 


= Config. files specifying schedule are loaded onto each network device. 
¢ Eliminating contention = no arbitration 
= Decentralized synchronization process establishes a global time base. 
= Devices reference time to dispatch messages at predetermined instances 


= Schedule guarantees no contention between TT frames. 
= Latency < 12.5 us/switch, < 1s jitter, no arbitration 


Note that controlling the jitter dramatically lowers latency compared to 
asynchronous RC traffic. A large portion of latency is the jitter! 
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TTEthernet overcomes difficulties in realizing an IMA architecture by providing 
three distinct traffic classes covering the full spectrum of criticality levels. 


Bandwidth ear Fa 
was Wialealae)ate)ersme(-1-1aanlialisiile 
Utilization messaging via Time- 
Low Triggered traffic w/ TDMA 
Partitioning (SAE AS6802) AS Merieeus 
3 ™ deterministic messaging 
y via Rate-Constrained 
9% diagnostics and traffic (ARINC 664- 
configuration (IEEE 802.3) - p?) 
18% high-definition video 
rar) streaming (IEEE 802.3) 
r= 
Oo 
= . 
fal 9% real-time sensor 
oO network (ARINC 664) Traffic shaping 
= and policing 
© 18% real-time audio 
streaming (ARINC 664) 


36% hard real-time control 
loops and processing over 
Ethernet backbone (SAE 
AS6802) 
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° Priority-based partitioning: 3 traffic classes on 1 physical layer. 
= Messages forwarded: 1) as scheduled (TT), or 2) as priority allows (RC, BE). 
= Bandwidth is released if TT message is not sent in synchronous time slot. 
= Ensuring determinism in a mixed-criticality network: 

— Timely block: Prevents RC or BE transmission during TT slots (unless freed). 
— Shuffling: Higher priority message is queued until lower priority frame is sent. 


ms 
6ms cluster cycle 11 
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TTEthernet network partitioning reduces cascading faults b/w platforms w/o 
the need for complex fault isolation procedures at the application level. 


¢ Traffic classes provide hard fault containment in the network. 
= Guaranteed TT frame delivery regardless of asynchronous traffic patterns. 
= Communication schedule controls access of devices to network resources. 


Sender knows 
exact frame 
dispatch time. 


= Switches act as central 
bus guardians to protect 
against arbitrarily faulty 
end systems. 
— TT: acceptance window 
Receiver knows — RC: temporal distance 


exact frame 
arrival time. 1 ) 


— 
~ -— 
Switch knows 
forwarding time. 


End system 


End system 


TDMA bandwidth 
partitioning of 
critical traffic. 
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° Precision Time Protocol (PTP IEEE 1588-2008) 


Grandmaster Clock 


State-of-the-art Ethernet clock synchronization Selected via BMC 
algorithm in industrial applications. ~~ 
Improves over Network Time Protocol (NTP) > s:beta a 
through specialized network hardware for time- “4 
stamping and decoding (sub-us accuracy). 


Protocol can be at Ethernet or IP layers. 
Hierarchical master/slave arch. for distributing 7 Brock 
time-of-day and clock frequency information. ni 


Uses best master clock (BMC) algorithm to 
select grandmaster clock source. Le Le 
Built-in redundancy means that if clock CAD C 


Boundary Clock 


source fails, another is selected. 


End Devices 
(ordinary clocks) 13 
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° Time-Triggered Ethernet (SAE AS6802) 
= Based on the exchange of asynchronous Protocol Control Frames (PCFs). 
= Each component is assigned one of three roles (SC, SM, or CM). 


° Two Step Process (integration cycle) 


= SMs dispatch PCFs to CMs at same ean 
local time (drift = actually different!). wins. 


= CMs send PCFs to all SCs and SMs, - in = Mester 


which they use to correct local time. 
° Key Differences 


= Decentralized “master”. 
= No search for best clock. 


= Tolerates multiple faults. 7. — FP se Client Ne ea Master 


= No external wall clock. 14 
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° Directly alters Ethernet data link 
layer (L2). Does not add additional 
protocol layers. 


TCP/IP Model Network Stack 


Layer7 | Application | 


= iia 
° Traffic classes can coexist with layers. Network (IPv4,1Pv6) 


other L2 QoS enhancements §$ -—|/ ---~----=----- 3---£- 
(e.g. IEEE 802.1Q). IEEE 802.10 


virtual LANs 
Common higher level protocols | *“Ethernet:- 


(VLANs) 


1Gbit/s Layer 2 IEEE 802.3 Ethernet | 
(e.g. [Pv4, UDP) can be used Fhesnce Sete ee So al 
on top of TTEthernet’s data 
link layer Physicdllhik_—_—_ 
7 bytes 1 byte 6 bytes 6 bytes 2 bytes 46-1500 bytes 4bytes 12 bytes 
Steaearenern [ome] eee” [ess [Rage [ eosrores [es [ie 
(Classical Ethernet) Preamble |SFD Address Address (Length) Data Payload FCS Pe 
7 bytes 1 oka 4 byte: 2 “haa 6 bytes 2 bytes 46-1500 bytes 4 bytes 12 bytes 
1 


ARINC 664-P7 (RC) 
SAE AS6802 (TT) 
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> SAE AS6802 (TT) and ARINC 664p-7 (RC) use Virtual Links (VLs) 
to replace traditional MAC-based message delivery. 
= Static forwarding table associates VLs 
with switch output ports. 
= VLs emulate point-to-point wiring 
seen in federated architectures. 


VL8, TT 
(fixed latency) 


-v 
/ 
I x2 Redundant 
Network Planes 


VL12, RC 
(max latency) 


Predefined .—~” 


¢ Increase fault-tolerance with multiple QoS per VL 
parallel switches. (TT vs RC) 

¢ Redundancy mgmt. discards extra frames. 

¢ Dual-fault tolerant w/ three redundant 
channels and high integrity Cece 


SWITCH SWITCH SWITCH 


Sample TTEthernet Network 
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° Past efforts used classical Ethernet over vehicle backbone. 


= Load balancer acted as virtual flight processor eae ae, | 
IP, detecting failure and directing TX/RX. =p 
= Introduces single point of failure. f | 


= Can increase fault tolerance w/ VRPP 
or redundant load balancers. 


3% Relies on monitoring with BE Ethernet. ae Ethernet 


Failover w/ load balancer 


° Failover with deterministic Ethernet 
= Virtual link based delivery removes need for load balancer. 
— |dentical messages can be dispatched to multiple recipients simultaneously. 
= Means FC’s have access to same data = More seamless failover. 
= Can increase fault tolerance with redundant TTEthernet switches. 
= Schedule driven communication compliments flight software behavior. +7 
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¢ What is the Ascent Abort 2 Flight Test? 
= Launch Abort System (LAS) carries CM away from ascent booster. 
= Goal is to stress the capabilities of synchronized redundant control loop. 
= Conducted AA-2 flight test demo in May ‘15 Integrated Test at JSC. 


° Redundant Flight Computer Architecture 
= Three identical redundant flight computers (pc-linux). 
= Failover logic built into Core Flight Software System (CFS). 
= Synchronization over TTEthernet network (200HZ). 


= CFS included several genuine Orion fsw components: 
— Absolute Navigation (AbsNav) for Exploration Mission EM-1. 
— Service module abort, stochastic/optical navigation, and propellant balancing. 
ANTARES simulation integrated into Tricksim. 
— Official NASA Orion spacecraft assessment tool used by JSC’s GNC branch. 18 


Simulation 
> Environment 
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° AA-2- Unique Mission Requirements: 


= Message payload sizes from simulation up to 20,000 bytes. 
— Ethernet frame data length is limited to 1500 bytes. 


= Throughput rates up to 100Mbit/s per Ethernet link. 
= Comm. with classical Ethernet systems w/o separate network adaptor. 


° Extension to TTEthernet Library Nol eivanssera gah 
(Phoenix IP - data link layer): : 


= Implements IPv4 (RFC 791) and 
UDP (RFC 768) protocol layers. 


= Abstraction from DMA management. 1 
= Built in software = cross-platform. 
= Maximizes throughput (e.g. minimize , 

copies, parallel checksum summation). ——— prnsical tink ————————+ 


Application 
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65,507 octets supported by library (max UDP data length according to RFC 5405) 


TTE Extended Library oon 
+ TTE Services Message written to TTE buffer 


| Message sid 


Select frame 
for transmission 


Redundancy 
Management 
IPv4 hdr Z hdr [v4har [| 
Replicate 


One or more ay produced Ethernet frame 


Add IP Checksum 


Build Ethernet 
Header 


IPv4 Network Layer 


= 


Data Link Layer 


Ethernethor | 


Sequence Number (RC only) 


Update Ethernet 
source MAC w/ 
physical port ID 


Ethernet hdr 


one e ane — Frames transmitted on physical links 
Begs 


TTEthernet Extended Library TX protocol stack 21 
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Software-Level Network Stack 


RC Raw Ethernet Frame (payload = 1500 bytes) RC UDP Datagram w/ frag (payload = 8192 bytes) 
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TTEthernet Library Extension throughput analysis on PC-Linux HP Z400 
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Combine the concept of scheduling the execution of CFS apps with 
the scheduling of the TTEthernet network. 

¢ Drives FSW execution off cluster cycle. 
¢ Can have deterministic scheduler even ay Gate Ge 


a (lee ee: [eer] [m7] 


B 
-_—___  """""""""""- 


3ms 3ms 


on limited hardware. 
a c falimlro|pe [ees 


2ms 
6ms cluster cycle 


Cluster Cycle Period = LCM of all TT comm periods in sync domain 


AFDX 
Endsystem : Minor Frame Period = (Cluster Cycle Period) x N 
TTEthe Hard real-time controls 
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° Message-based Synchronization 
= Master/Slave architecture. 
= Master computer drives CFS schedule 
off internal or network based timer. 
= Highest-priority FC commands lower 
priority machines to move b/w slots. 


° Network-based Synchronization 


= Distributed architecture. 


= Each FC drives CFS schedule off 
network interrupts (e.g. cluster cycle). 


= Cluster period is a global property. 
Interrupts are generated on each 
Trigger slot transition every N 


machine simultaneously. cluster cycle interrupts 
Network-based sync in AA-2FSW 24 


Major Frame Major Frame 


Rate-Constrained Traffic 


Andrew Loveless, NASA/JSC 
la tana 


Shaping the Future of Aerospace 


male lalme@ze)an)elel(cis 
Linux PC (64-bit) 


eer UD 

40Hz communication rate ae 

between FCs and sim 18388 bytes 
RC-Eth 


Flight Computer es gees | CK Simulation 
Linux PC (64-bit) Linux PC (64-bit) 


Failover to consecutive slot 
guaranteed 


TT-Eth 


IPv4 and UDP layers 
used for flight control 


loop 
male lajm@xeyan) evel (ais 
Linux PC (64-bit) 


RC traffic used for sync 
between computers 


Final setup for May ‘15 Integrated Test at JSC (AA-2 simulation) 25 
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